30 Days Readmission Predicition with Diagnosis Features¶

1 Overview and Background¶

With the development of modern large scale computing technologies, machine learing techniques have been increasingly applied to large scale health care datasets, such as clinical data and biological data. Developing and optimizing health-related systems are no doubt promising as our human beings' health conditions always come first. Among numerous research topics, predicting hospital 30-Day readmission is of our interests. If we are able to accurately predict readmission, we can identify patients at high risk and prevent early discharging them. As a result, patients will be given more precision care and hospitals can better manage resources, reduce waste room and save costs. Existing works have already explored this topic [1-4]. Before designing the model, extracting most useful features is of great importance. By far, the features that have been taken into consideration includes, demographic characteristics: age, gender, race etc; lab results, chart events etc. monitored in ICU [1]; electronic health record data: length of stay, number of admissions, admission type etc. [2,3]. Machine learning techniques that have been used include random forest [1], neural networks [4] etc. Our project goal is to extract relevant features from MIMIC database and design classification models to optimize performance.

References:

[1] Yaron Blinder. Predicting 30-day ICU readmissions from the MIMIC-III database. https://github.com/YaronBlinder/MIMIC-III_readmission. 2017.

[2] Oanh Kieu Nguyen et al. Predicting all-cause readmissions using electronic health record data from the entire hospitalization: model development and comparison". In: Journal of hospital medicine 11.7 (2016), pp. 473-480.

[3] Frida Kareliusson, Lina De Geer, and Anna Oscarsson Tibblin. Risk prediction of ICU readmission in a mixed surgical and medical population". In: Journal of intensive care 3.1 (2015), p. 30.

[4] Ricardo Bento Afonso. Feature Extraction and Selection for Prediction of ICU Patient's Readmission Using Artificial Neural Networks". In: (2013).

2 Data Processing¶

Genrate lables: For each patient, we computed the intervals between his two sequent admissions. If the interval is shorter than 30 days, the readmission label for the encounter is 1; otherwise the readmission label is 0.

Create feature matrix: We joined the diagnosis table and the ICD code table to obtain the diagnosis for each encounter. Since the diagnosis are categorical variables, we converted the categorical variable into dummy variables. We also added some other predictive features, such as genders and admission types.

import sqlite3 
import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import BaggingClassifier

from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import StratifiedKFold

from sklearn.metrics import confusion_matrix

import time
import math

import warnings
warnings.filterwarnings('ignore')

from imblearn.over_sampling import RandomOverSampler
from imblearn.under_sampling import RandomUnderSampler

plt.rcParams['figure.dpi'] = 300

conn = sqlite3.connect('C:/Users/culiubaicaiya/Downloads/bme590 course note/data/MIMIC.db')

pd.set_option('display.max_columns', None)

table_df = pd.read_sql("SELECT * FROM sqlite_master where type='table'", conn)

table_df

def display_table(table_name):
    print(table_name)
    display(pd.read_sql('SELECT * FROM {} LIMIT 5'.format(table_name), conn))

table_list = table_df['tbl_name'].values

for table_name in table_list:
    display_table(table_name)

CHARTEVENTS

PROCEDUREEVENTS_MV

CALLOUT

D_CPT

D_ITEMS

CAREGIVERS

MICROBIOLOGYEVENTS

LABEVENTS

INPUTEVENTS_CV

ADMISSIONS

2.1 Generate Labels¶

readmit_df = pd.read_sql('''SELECT * FROM
                           (SELECT SUBJECT_ID, HADM_ID, ADMITTIME, DISCHTIME, 
                                   RANK() OVER(PARTITION BY SUBJECT_ID ORDER BY ADMITTIME) AS ADMITTIME_RANK 
                            FROM ADMISSIONS) a
                            LEFT JOIN 
                           (SELECT SUBJECT_ID, MAX(ADMITTIME_RANK) AS MAX_ADMITTIME_RANK FROM
                           (SELECT SUBJECT_ID, HADM_ID, ADMITTIME, DISCHTIME, 
                                   RANK() OVER(PARTITION BY SUBJECT_ID ORDER BY ADMITTIME) AS ADMITTIME_RANK 
                            FROM ADMISSIONS)
                            GROUP BY SUBJECT_ID) b
                            ON a.SUBJECT_ID = b.SUBJECT_ID''', conn)

readmit_df[['ADMITTIME','DISCHTIME']] = readmit_df[['ADMITTIME','DISCHTIME']].apply(lambda x: pd.to_datetime(x, format='%Y-%m-%d %H:%M:%S'))

next_admittime = readmit_df['ADMITTIME'].values[1:]
next_admittime = np.append(next_admittime, next_admittime[0])
readmit_df['NEXT_ADMITTIME'] = next_admittime

readmit_df['30_READMIT'] = -1

readmit_df.loc[readmit_df['ADMITTIME_RANK'] == readmit_df['MAX_ADMITTIME_RANK'], '30_READMIT'] = 0

readmit_sub_df = readmit_df[readmit_df['30_READMIT']==-1]
interval = (readmit_sub_df['NEXT_ADMITTIME'] - readmit_sub_df['DISCHTIME']).dt.days.values
readmit_df.loc[readmit_df['30_READMIT']==-1, 'INTERVAL'] = interval 

readmit_df.loc[readmit_df['INTERVAL']<=30, '30_READMIT'] = 1
readmit_df.loc[readmit_df['INTERVAL']>30, '30_READMIT'] = 0

readmit_df

readmit_label_df = readmit_df[['HADM_ID', '30_READMIT']]

readmit_label_df.head()

readmit_label_df.shape

(58976, 2)

2.2 Create Feature Matrix¶

feat_sub1_df = pd.read_sql('''SELECT a.HADM_ID, b.SHORT_TITLE
                         FROM DIAGNOSES_ICD a
                         INNER JOIN D_ICD_DIAGNOSES b on a.ICD9_CODE = b.ICD9_CODE
                            
''',conn)

feat_sub1_df = feat_sub1_df.groupby(['HADM_ID', 'SHORT_TITLE']).size().unstack()

feat_sub1_df = feat_sub1_df.notnull()
feat_sub1_df = feat_sub1_df * 1
feat_sub1_df = feat_sub1_df.apply(pd.to_numeric)

feat_sub1_df.head()

feat_sub1_df.shape

(58925, 6770)

feat_sub2_df = pd.read_sql('''SELECT a.HADM_ID, a.ADMISSION_TYPE, b.gender 
                              FROM ADMISSIONS a 
                              INNER JOIN PATIENTS b on a.SUBJECT_ID = b.SUBJECT_ID''',conn)

feat_sub2_df = pd.get_dummies(feat_sub2_df)

feat_sub2_df.head()

feat_sub2_df.shape

(58976, 7)

feat_df = pd.merge(feat_sub1_df, feat_sub2_df, left_on=['HADM_ID'], right_on=['HADM_ID'], how='inner')

feat_df.head()

feat_df.shape

(58925, 6777)

2.3 Create Design Matrix¶

design_matrix = pd.merge(readmit_label_df, feat_df, left_on =['HADM_ID'], right_on=['HADM_ID'], how='inner')

design_matrix.shape

(58925, 6778)

design_matrix.head()

3 Data Exploration¶

We visualized the readmission proportion of the two datasets. Apparently, the datasets have highly imbalanced classes.

X = design_matrix.iloc[:, 2:].values

X

array([[0, 0, 0, ..., 0, 0, 1],
       [0, 0, 0, ..., 0, 0, 1],
       [0, 0, 0, ..., 0, 1, 0],
       ...,
       [0, 0, 0, ..., 0, 1, 0],
       [0, 0, 0, ..., 0, 1, 0],
       [0, 0, 0, ..., 0, 1, 0]], dtype=int32)

X.shape

(58925, 6776)

print('number of instance : the number features = {}'.format(X.shape[0] / X.shape[1]) )

number of instance : the number features = 8.696133412042503

y = design_matrix['30_READMIT'].values

H1_num = len(y[y==1])
H0_num = len(y[y==0])
total_num = len(y)

print('number of data instance in class1:{}'.format(H1_num))

number of data instance in class1:3454

labels = '30Days Readmission-Y', '30Days Readmission-N'
sizes = [H1_num/total_num*100, H0_num/total_num*100]
colors = ['pink', 'skyblue']
explode = (.05, 0)
fig = plt.figure(figsize=(6,6))
plt.pie(sizes, explode=explode, colors=colors, autopct='%1.1f%%', startangle=90)
plt.legend(labels, fontsize='small', loc='center left', bbox_to_anchor=(1, 0, 0.5, 1))

<matplotlib.legend.Legend at 0x24ae217e1d0>

4 Modeling¶

We first randomly assigned the data instances into different folds to prepare for cross validation. For preprocessing, we employed down sampling and over sampling methods to relieve the class imbalance problem. Since the dataset has extremely high dimensions (the ratio of the number of instances to the number features is approximately 8:1), we used l1 penalty for feature selection. Then we tested logistic regression and decision tree. To increase the performance, we applied the ensemble methods, including bagging logitic regression and random forest. We also designed a neural network for classification. Finally, we evaluated the cross validation performance and tuned parameters to optimize the performance.

from IPython.display import Image
PATH = "./pic/"
Image(filename = PATH + "pipeline.jpg", width=800, height=400)

4.1 Logistic Regression¶

4.1.1 Over - Sampling¶

ros = RandomOverSampler(random_state=42)

skf = StratifiedKFold(n_splits=5, shuffle=False, random_state=0)
y_true = []
y_pred = []

for train_index, test_index in skf.split(X, y):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    X_train_ros, y_train_ros = ros.fit_resample(X_train, y_train)
    
    lr_clf = LogisticRegression(penalty='l1', random_state=0, solver='liblinear') # l1-based feature selection
    lr_clf.fit(X_train_ros, y_train_ros)
    y_pred_sub = lr_clf.predict(X_test)
    y_true = np.append(y_true, y_test)
    y_pred = np.append(y_pred, y_pred_sub)
    
lr_os_conf = confusion_matrix(y_true, y_pred)
tn, fp, fn, tp = lr_os_conf.ravel()
lr_os_sen = tp/(tp+fn)
lr_os_spe = tn/(tn+fp)
lr_os_prec = tp/(tp+fp)
lr_os_acc = (tp+tn)/(tp+tn+fp+fn)

print ('Logistic Regression, Over_Sampling')
print()
print('confusion matrix:')
print(lr_os_conf)
print('sensitivity score:{:.4}'.format(lr_os_sen))
print('specificity score:{:.4}'.format(lr_os_spe))
print('precision score:{:.4}'.format(lr_os_prec))
print('accuracy score:{:.4}'.format(lr_os_acc))

Logistic Regression, Over_Sampling

confusion matrix:
[[42818 12653]
 [ 2128  1326]]
sensitivity score:0.3839
specificity score:0.7719
precision score:0.09486
accuracy score:0.7492

4.1.2 Under - Sampling¶

rus = RandomUnderSampler(random_state=42)

skf = StratifiedKFold(n_splits=5, shuffle=False, random_state=0)
y_true = []
y_pred = []

for train_index, test_index in skf.split(X, y):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    X_train_rus, y_train_rus = rus.fit_resample(X_train, y_train)
    
    lr_clf = LogisticRegression(penalty='l1', random_state=0, solver='liblinear') # l1-based feature selection
    lr_clf.fit(X_train_rus, y_train_rus)
    y_pred_sub = lr_clf.predict(X_test)
    y_true = np.append(y_true, y_test)
    y_pred = np.append(y_pred, y_pred_sub)
    
lr_us_conf = confusion_matrix(y_true, y_pred)
tn, fp, fn, tp = lr_us_conf.ravel()
lr_us_sen = tp/(tp+fn)
lr_us_spe = tn/(tn+fp)
lr_us_prec = tp/(tp+fp)
lr_us_acc = (tp+tn)/(tp+tn+fp+fn)

print ('Logistic Regression, Under_Sampling')
print()
print('confusion matrix:')
print(lr_us_conf)
print('sensitivity score:{:.4}'.format(lr_us_sen))
print('specificity score:{:.4}'.format(lr_us_spe))
print('precision score:{:.4}'.format(lr_us_prec))
print('accuracy score:{:.4}'.format(lr_us_acc))

Logistic Regression, Under_Sampling

confusion matrix:
[[35815 19656]
 [ 1462  1992]]
sensitivity score:0.5767
specificity score:0.6457
precision score:0.09202
accuracy score:0.6416

4.2 Bagging Logistic Regression¶

4.2.2 Over- Sampling¶

ros = RandomOverSampler(random_state=42)

skf = StratifiedKFold(n_splits=5, shuffle=False, random_state=0)
y_true = []
y_pred = []

for train_index, test_index in skf.split(X, y):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    X_train_ros, y_train_ros = ros.fit_resample(X_train, y_train)
    
    base_estimator = LogisticRegression(penalty='l1', random_state=0, solver='liblinear') # l1-based feature selection
    max_features = int(math.sqrt(X_train.shape[1]))
    blr_clf = BaggingClassifier(base_estimator=base_estimator, n_estimators=100, max_features=max_features, random_state=0)
    blr_clf.fit(X_train_ros, y_train_ros)
    y_pred_sub = blr_clf.predict(X_test)
    y_true = np.append(y_true, y_test)
    y_pred = np.append(y_pred, y_pred_sub)
    
blr_os_conf = confusion_matrix(y_true, y_pred)
tn, fp, fn, tp = blr_os_conf.ravel()
blr_os_sen = tp/(tp+fn)
blr_os_spe = tn/(tn+fp)
blr_os_prec = tp/(tp+fp)
blr_os_acc = (tp+tn)/(tp+tn+fp+fn)

print ('Bagging Logistic Regression, Over_Sampling')
print()
print('confusion matrix:')
print(blr_os_conf)
print('sensitivity score:{:.4}'.format(blr_os_sen))
print('specificity score:{:.4}'.format(blr_os_spe))
print('precision score:{:.4}'.format(blr_os_prec))
print('accuracy score:{:.4}'.format(blr_os_acc))

Bagging Logistic Regression, Over_Sampling

confusion matrix:
[[38344 17127]
 [ 1668  1786]]
sensitivity score:0.5171
specificity score:0.6912
precision score:0.09443
accuracy score:0.681

4.2.2 Under - Sampling¶

rus = RandomUnderSampler(random_state=42)

skf = StratifiedKFold(n_splits=5, shuffle=False, random_state=0)
y_true = []
y_pred = []

for train_index, test_index in skf.split(X, y):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    X_train_rus, y_train_rus = rus.fit_resample(X_train, y_train)
    
    base_estimator = LogisticRegression(penalty='l1', random_state=0, solver='liblinear') # l1-based feature selection
    max_features = int(math.sqrt(X_train.shape[1]))
    blr_clf = BaggingClassifier(base_estimator=base_estimator, n_estimators=100, max_features=max_features, random_state=0)
    blr_clf.fit(X_train_rus, y_train_rus)
    y_pred_sub = blr_clf.predict(X_test)
    y_true = np.append(y_true, y_test)
    y_pred = np.append(y_pred, y_pred_sub)
    
blr_us_conf = confusion_matrix(y_true, y_pred)
tn, fp, fn, tp = blr_us_conf.ravel()
blr_us_sen = tp/(tp+fn)
blr_us_spe = tn/(tn+fp)
blr_us_prec = tp/(tp+fp)
blr_us_acc = (tp+tn)/(tp+tn+fp+fn)

print ('Bagging Logistic Regression, Under_Sampling')
print()
print('confusion matrix:')
print(blr_us_conf)
print('sensitivity score:{:.4}'.format(blr_us_sen))
print('specificity score:{:.4}'.format(blr_us_spe))
print('precision score:{:.4}'.format(blr_us_prec))
print('accuracy score:{:.4}'.format(blr_us_acc))

Bagging Logistic Regression, Under_Sampling

confusion matrix:
[[33597 21874]
 [ 1364  2090]]
sensitivity score:0.6051
specificity score:0.6057
precision score:0.08721
accuracy score:0.6056

4.3 Decision Tree¶

skf = StratifiedKFold(n_splits=5, shuffle=False, random_state=0)
y_true = []
y_pred = []

for train_index, test_index in skf.split(X, y):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    # l1-based feature selection
    logit_clf = LogisticRegression(penalty='l1', random_state=0, solver='liblinear')
    logit_clf.fit(X_train, y_train)
    select_model = SelectFromModel(logit_clf)
    select_model.fit(X_train, y_train)
    X_train_new = select_model.transform(X_train)
    X_test_new = select_model.transform(X_test)
    
    dt_clf = DecisionTreeClassifier(max_depth=3, random_state=0, class_weight='balanced')
    dt_clf.fit(X_train_new, y_train)
    y_pred_sub = dt_clf.predict(X_test_new)
    y_true = np.append(y_true, y_test)
    y_pred = np.append(y_pred, y_pred_sub)

dt_conf = confusion_matrix(y_true, y_pred)
tn, fp, fn, tp = dt_conf.ravel()
dt_sen = tp/(tp+fn)
dt_spe = tn/(tn+fp)
dt_prec = tp/(tp+fp)
dt_acc = (tp+tn)/(tp+tn+fp+fn)

print ('Decision Tree')
print()
print('confusion matrix:')
print(dt_conf)
print('sensitivity score:{:.4}'.format(dt_sen))
print('specificity score:{:.4}'.format(dt_spe))
print('precision score:{:.4}'.format(dt_prec))
print('accuracy score:{:.4}'.format(dt_acc))

Decision Tree

confusion matrix:
[[36419 19052]
 [ 1944  1510]]
sensitivity score:0.4372
specificity score:0.6565
precision score:0.07344
accuracy score:0.6437

4.4 Random Forest¶

skf = StratifiedKFold(n_splits=5, shuffle=False, random_state=0)
y_true = []
y_pred = []

for train_index, test_index in skf.split(X, y):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    # l1-based feature selection
    logit_clf = LogisticRegression(penalty='l1', random_state=0, solver='liblinear')
    logit_clf.fit(X_train, y_train)
    select_model = SelectFromModel(logit_clf)
    select_model.fit(X_train, y_train)
    X_train_new = select_model.transform(X_train)
    X_test_new = select_model.transform(X_test)
    
    rf_clf = RandomForestClassifier(n_estimators=100, max_depth=3, random_state=0, class_weight='balanced')
    rf_clf.fit(X_train_new, y_train)
    y_pred_sub = rf_clf.predict(X_test_new)
    y_true = np.append(y_true, y_test)
    y_pred = np.append(y_pred, y_pred_sub)

rf_conf = confusion_matrix(y_true, y_pred)
tn, fp, fn, tp = rf_conf.ravel()
rf_sen = tp/(tp+fn)
rf_spe = tn/(tn+fp)
rf_prec = tp/(tp+fp)
rf_acc = (tp+tn)/(tp+tn+fp+fn)

print ('Random Forest')
print()
print('confusion matrix:')
print(rf_conf)
print('sensitivity score:{:.4}'.format(rf_sen))
print('specificity score:{:.4}'.format(rf_spe))
print('precision score:{:.4}'.format(rf_prec))
print('accuracy score:{:.4}'.format(rf_acc))

Random Forest

confusion matrix:
[[34536 20935]
 [ 1397  2057]]
sensitivity score:0.5955
specificity score:0.6226
precision score:0.08947
accuracy score:0.621

4.5 Neural Network (Deep Learning)¶

from keras import backend as K
from keras.models import Sequential
from keras.layers import Activation, Dense, Dropout
seed_value= 0
# 1. Set `PYTHONHASHSEED` environment variable at a fixed value
import os
os.environ['PYTHONHASHSEED']=str(seed_value)
# 2. Set `python` built-in pseudo-random generator at a fixed value
import random
random.seed(seed_value)
# 3. Set `numpy` pseudo-random generator at a fixed value
np.random.seed(seed_value)
# 4. Set `tensorflow` pseudo-random generator at a fixed value
import tensorflow as tf
tf.set_random_seed(seed_value)
os.environ['KMP_DUPLICATE_LIB_OK']='True'

Using TensorFlow backend.

skf = StratifiedKFold(n_splits=5, shuffle=False, random_state=0)
y_true = []
y_pred = []

for train_index, test_index in skf.split(X, y):
    model = Sequential()
    model.add(Dense(512, activation='relu'))
    model.add(Dropout(0.2))
    model.add(Dense(256, activation='relu'))
    model.add(Dropout(0.2))
    model.add(Dense(8, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
    class_weight = {0: 1., 1: 1400.}
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    
    model.fit(X_train, y_train, batch_size=1000, epochs=10, validation_data=(X_test, y_test), verbose=1, class_weight=class_weight)
    y_pred_sub = model.predict(X_test).flatten()
    y_pred_sub[y_pred_sub >= 0.5] = 1
    y_pred_sub[y_pred_sub < 0.5] = 0
    y_true = np.append(y_true, y_test)
    y_pred = np.append(y_pred, y_pred_sub)
    
nn_conf = confusion_matrix(y_true, y_pred)
tn, fp, fn, tp = nn_conf.ravel()
nn_sen = tp/(tp+fn)
nn_spe = tn/(tn+fp)
nn_prec = tp/(tp+fp)
nn_acc = (tp+tn)/(tp+tn+fp+fn)

print ('Neural Network(Deep Learning)')
print()
print('confusion matrix:')
print(nn_conf)
print('sensitivity score:{:.4}'.format(nn_sen))
print('specificity score:{:.4}'.format(nn_spe))
print('precision score:{:.4}'.format(nn_prec))
print('accuracy score:{:.4}'.format(nn_acc))

WARNING:tensorflow:From C:\Users\culiubaicaiya\Anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From C:\Users\culiubaicaiya\Anaconda3\lib\site-packages\keras\backend\tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From C:\Users\culiubaicaiya\Anaconda3\lib\site-packages\tensorflow\python\ops\math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Train on 47139 samples, validate on 11786 samples
Epoch 1/10
47139/47139 [==============================] - 9s 181us/step - loss: 13.5791 - acc: 0.0679 - val_loss: 3.9564 - val_acc: 0.0586
Epoch 2/10
47139/47139 [==============================] - 8s 176us/step - loss: 4.8857 - acc: 0.0586 - val_loss: 3.6194 - val_acc: 0.0586
Epoch 3/10
47139/47139 [==============================] - 8s 164us/step - loss: 4.2696 - acc: 0.0586 - val_loss: 3.4620 - val_acc: 0.0586
Epoch 4/10
47139/47139 [==============================] - 8s 174us/step - loss: 3.9094 - acc: 0.0586 - val_loss: 3.1396 - val_acc: 0.0586
Epoch 5/10
47139/47139 [==============================] - 8s 175us/step - loss: 3.6077 - acc: 0.0586 - val_loss: 3.0530 - val_acc: 0.0586
Epoch 6/10
47139/47139 [==============================] - 8s 177us/step - loss: 3.3209 - acc: 0.0586 - val_loss: 2.8240 - val_acc: 0.0586
Epoch 7/10
47139/47139 [==============================] - 8s 172us/step - loss: 3.0093 - acc: 0.0586 - val_loss: 2.6891 - val_acc: 0.0586
Epoch 8/10
47139/47139 [==============================] - 8s 173us/step - loss: 2.6737 - acc: 0.0788 - val_loss: 2.4193 - val_acc: 0.1123
Epoch 9/10
47139/47139 [==============================] - 8s 172us/step - loss: 2.2899 - acc: 0.2204 - val_loss: 2.1411 - val_acc: 0.1890
Epoch 10/10
47139/47139 [==============================] - 8s 167us/step - loss: 1.8784 - acc: 0.3966 - val_loss: 1.8644 - val_acc: 0.3245
Train on 47140 samples, validate on 11785 samples
Epoch 1/10
47140/47140 [==============================] - 9s 186us/step - loss: 12.3106 - acc: 0.0639 - val_loss: 3.6580 - val_acc: 0.0586
Epoch 2/10
47140/47140 [==============================] - 8s 180us/step - loss: 4.8109 - acc: 0.0586 - val_loss: 3.5442 - val_acc: 0.0586
Epoch 3/10
47140/47140 [==============================] - 9s 182us/step - loss: 4.1530 - acc: 0.0586 - val_loss: 3.2732 - val_acc: 0.0586
Epoch 4/10
47140/47140 [==============================] - 8s 174us/step - loss: 3.7667 - acc: 0.0586 - val_loss: 3.0952 - val_acc: 0.0586
Epoch 5/10
47140/47140 [==============================] - 8s 168us/step - loss: 3.4117 - acc: 0.0586 - val_loss: 2.8657 - val_acc: 0.0586
Epoch 6/10
47140/47140 [==============================] - 8s 167us/step - loss: 3.0953 - acc: 0.0586 - val_loss: 2.7789 - val_acc: 0.0586
Epoch 7/10
47140/47140 [==============================] - 8s 175us/step - loss: 2.7058 - acc: 0.0586 - val_loss: 2.3826 - val_acc: 0.0586
Epoch 8/10
47140/47140 [==============================] - 9s 183us/step - loss: 2.2895 - acc: 0.0593 - val_loss: 2.0849 - val_acc: 0.0586
Epoch 9/10
47140/47140 [==============================] - 9s 184us/step - loss: 1.8573 - acc: 0.1497 - val_loss: 1.7146 - val_acc: 0.3012
Epoch 10/10
47140/47140 [==============================] - 8s 166us/step - loss: 1.5300 - acc: 0.5224 - val_loss: 1.6263 - val_acc: 0.4215
Train on 47140 samples, validate on 11785 samples
Epoch 1/10
47140/47140 [==============================] - 9s 183us/step - loss: 13.2882 - acc: 0.0725 - val_loss: 4.0586 - val_acc: 0.0586
Epoch 2/10
47140/47140 [==============================] - 8s 165us/step - loss: 4.8585 - acc: 0.0586 - val_loss: 3.7015 - val_acc: 0.0586
Epoch 3/10
47140/47140 [==============================] - 8s 168us/step - loss: 4.2389 - acc: 0.0586 - val_loss: 3.4246 - val_acc: 0.0586
Epoch 4/10
47140/47140 [==============================] - 8s 168us/step - loss: 3.8594 - acc: 0.0586 - val_loss: 3.1617 - val_acc: 0.0586
Epoch 5/10
47140/47140 [==============================] - 8s 169us/step - loss: 3.5175 - acc: 0.0586 - val_loss: 2.9755 - val_acc: 0.0586
Epoch 6/10
47140/47140 [==============================] - 8s 171us/step - loss: 3.1620 - acc: 0.0586 - val_loss: 2.6978 - val_acc: 0.0586
Epoch 7/10
47140/47140 [==============================] - 8s 170us/step - loss: 2.8004 - acc: 0.0586 - val_loss: 2.5206 - val_acc: 0.0586
Epoch 8/10
47140/47140 [==============================] - 8s 170us/step - loss: 2.4092 - acc: 0.0586 - val_loss: 2.2476 - val_acc: 0.0586
Epoch 9/10
47140/47140 [==============================] - 8s 172us/step - loss: 2.0536 - acc: 0.0586 - val_loss: 2.0710 - val_acc: 0.0586
Epoch 10/10
47140/47140 [==============================] - 8s 172us/step - loss: 1.7317 - acc: 0.0586 - val_loss: 1.7350 - val_acc: 0.0586
Train on 47140 samples, validate on 11785 samples
Epoch 1/10
47140/47140 [==============================] - 9s 189us/step - loss: 11.8339 - acc: 0.0702 - val_loss: 4.1645 - val_acc: 0.0586
Epoch 2/10
47140/47140 [==============================] - 9s 181us/step - loss: 4.7597 - acc: 0.0586 - val_loss: 3.7253 - val_acc: 0.0586
Epoch 3/10
47140/47140 [==============================] - 8s 177us/step - loss: 4.1553 - acc: 0.0586 - val_loss: 3.3748 - val_acc: 0.0586
Epoch 4/10
47140/47140 [==============================] - 8s 179us/step - loss: 3.7888 - acc: 0.0586 - val_loss: 3.0867 - val_acc: 0.0586
Epoch 5/10
47140/47140 [==============================] - 8s 175us/step - loss: 3.4906 - acc: 0.0586 - val_loss: 2.7386 - val_acc: 0.0586
Epoch 6/10
47140/47140 [==============================] - 8s 171us/step - loss: 3.1978 - acc: 0.0586 - val_loss: 2.4845 - val_acc: 0.0586
Epoch 7/10
47140/47140 [==============================] - 8s 171us/step - loss: 2.9263 - acc: 0.0586 - val_loss: 2.2732 - val_acc: 0.0586
Epoch 8/10
47140/47140 [==============================] - 8s 174us/step - loss: 2.6281 - acc: 0.0586 - val_loss: 2.1279 - val_acc: 0.0586
Epoch 9/10
47140/47140 [==============================] - 8s 174us/step - loss: 2.3394 - acc: 0.0779 - val_loss: 1.9519 - val_acc: 0.1417
Epoch 10/10
47140/47140 [==============================] - 8s 173us/step - loss: 2.0906 - acc: 0.2613 - val_loss: 1.7048 - val_acc: 0.2835
Train on 47141 samples, validate on 11784 samples
Epoch 1/10
47141/47141 [==============================] - 9s 189us/step - loss: 12.7684 - acc: 0.0619 - val_loss: 4.3753 - val_acc: 0.0586
Epoch 2/10
47141/47141 [==============================] - 8s 171us/step - loss: 4.8921 - acc: 0.0586 - val_loss: 3.8949 - val_acc: 0.0586
Epoch 3/10
47141/47141 [==============================] - 8s 171us/step - loss: 4.2689 - acc: 0.0586 - val_loss: 3.3750 - val_acc: 0.0586
Epoch 4/10
47141/47141 [==============================] - 8s 173us/step - loss: 3.8983 - acc: 0.0586 - val_loss: 3.0226 - val_acc: 0.0586
Epoch 5/10
47141/47141 [==============================] - 8s 172us/step - loss: 3.5928 - acc: 0.0586 - val_loss: 2.8039 - val_acc: 0.0586
Epoch 6/10
47141/47141 [==============================] - 8s 171us/step - loss: 3.2942 - acc: 0.0586 - val_loss: 2.4744 - val_acc: 0.0586
Epoch 7/10
47141/47141 [==============================] - 8s 172us/step - loss: 2.9852 - acc: 0.0586 - val_loss: 2.3063 - val_acc: 0.0586
Epoch 8/10
47141/47141 [==============================] - 8s 173us/step - loss: 2.6092 - acc: 0.0586 - val_loss: 1.8799 - val_acc: 0.0586
Epoch 9/10
47141/47141 [==============================] - 8s 169us/step - loss: 2.2351 - acc: 0.1476 - val_loss: 1.6062 - val_acc: 0.2745
Epoch 10/10
47141/47141 [==============================] - 8s 170us/step - loss: 1.8270 - acc: 0.3981 - val_loss: 1.1697 - val_acc: 0.5667
Neural Network(Deep Learning)

confusion matrix:
[[16938 38533]
 [  890  2564]]
sensitivity score:0.7423
specificity score:0.3053
precision score:0.06239
accuracy score:0.331

rus = RandomUnderSampler(random_state=42)

skf = StratifiedKFold(n_splits=5, shuffle=False, random_state=0)
y_true = []
y_pred = []

for train_index, test_index in skf.split(X, y):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    X_train_rus, y_train_rus = rus.fit_resample(X_train, y_train)
    
    model = Sequential()
    model.add(Dense(512, activation='relu'))
    model.add(Dropout(0.2))
    model.add(Dense(256, activation='relu'))
    model.add(Dropout(0.2))
    model.add(Dense(8, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
    class_weight = class_weight = {0: 1., 1: 1.}
    model.fit(X_train_rus, y_train_rus, batch_size=500, epochs=10, validation_data=(X_test, y_test), verbose=1, class_weight=class_weight)
    y_pred_sub = model.predict(X_test).flatten()
    y_pred_sub[y_pred_sub >= 0.5] = 1
    y_pred_sub[y_pred_sub < 0.5] = 0
    y_true = np.append(y_true, y_test)
    y_pred = np.append(y_pred, y_pred_sub)
    
nn_us_conf = confusion_matrix(y_true, y_pred)
tn, fp, fn, tp = nn_us_conf.ravel()
nn_us_sen = tp/(tp+fn)
nn_us_spe = tn/(tn+fp)
nn_us_prec = tp/(tp+fp)
nn_us_acc = (tp+tn)/(tp+tn+fp+fn)

print ('Neural Network(Deep Learning), Under-Sampling')
print()
print('confusion matrix:')
print(nn_us_conf)
print('sensitivity score:{:.4}'.format(nn_us_sen))
print('specificity score:{:.4}'.format(nn_us_spe))
print('precision score:{:.4}'.format(nn_us_prec))
print('accuracy score:{:.4}'.format(nn_us_acc))

Train on 5526 samples, validate on 11786 samples
Epoch 1/10
5526/5526 [==============================] - 3s 575us/step - loss: 0.6777 - acc: 0.5847 - val_loss: 0.6083 - val_acc: 0.6934
Epoch 2/10
5526/5526 [==============================] - 2s 371us/step - loss: 0.5998 - acc: 0.6911 - val_loss: 0.5464 - val_acc: 0.7447
Epoch 3/10
5526/5526 [==============================] - 2s 388us/step - loss: 0.5026 - acc: 0.7649 - val_loss: 0.6650 - val_acc: 0.6674
Epoch 4/10
5526/5526 [==============================] - 2s 393us/step - loss: 0.3813 - acc: 0.8393 - val_loss: 0.7568 - val_acc: 0.6254
Epoch 5/10
5526/5526 [==============================] - 2s 387us/step - loss: 0.2627 - acc: 0.9043 - val_loss: 0.6835 - val_acc: 0.6946
Epoch 6/10
5526/5526 [==============================] - 2s 387us/step - loss: 0.1660 - acc: 0.9448 - val_loss: 0.8582 - val_acc: 0.6601
Epoch 7/10
5526/5526 [==============================] - 2s 398us/step - loss: 0.1031 - acc: 0.9682 - val_loss: 1.0694 - val_acc: 0.6323
Epoch 8/10
5526/5526 [==============================] - 2s 389us/step - loss: 0.0689 - acc: 0.9821 - val_loss: 1.3240 - val_acc: 0.5888
Epoch 9/10
5526/5526 [==============================] - 2s 393us/step - loss: 0.0493 - acc: 0.9872 - val_loss: 1.1563 - val_acc: 0.6660
Epoch 10/10
5526/5526 [==============================] - 2s 388us/step - loss: 0.0458 - acc: 0.9852 - val_loss: 1.1461 - val_acc: 0.6724
Train on 5526 samples, validate on 11785 samples
Epoch 1/10
5526/5526 [==============================] - 3s 576us/step - loss: 0.6802 - acc: 0.5686 - val_loss: 0.6691 - val_acc: 0.6623
Epoch 2/10
5526/5526 [==============================] - 2s 378us/step - loss: 0.6003 - acc: 0.7029 - val_loss: 0.6029 - val_acc: 0.7146
Epoch 3/10
5526/5526 [==============================] - 2s 377us/step - loss: 0.4927 - acc: 0.7718 - val_loss: 0.7081 - val_acc: 0.6466
Epoch 4/10
5526/5526 [==============================] - 2s 377us/step - loss: 0.3672 - acc: 0.8484 - val_loss: 0.7207 - val_acc: 0.6691
Epoch 5/10
5526/5526 [==============================] - 2s 386us/step - loss: 0.2431 - acc: 0.9117 - val_loss: 1.0570 - val_acc: 0.5695
Epoch 6/10
5526/5526 [==============================] - 2s 392us/step - loss: 0.1590 - acc: 0.9441 - val_loss: 0.6287 - val_acc: 0.7620
Epoch 7/10
5526/5526 [==============================] - 2s 394us/step - loss: 0.1265 - acc: 0.9564 - val_loss: 1.1970 - val_acc: 0.6137
Epoch 8/10
5526/5526 [==============================] - 2s 386us/step - loss: 0.0854 - acc: 0.9721 - val_loss: 1.4631 - val_acc: 0.5672
Epoch 9/10
5526/5526 [==============================] - 2s 397us/step - loss: 0.0621 - acc: 0.9819 - val_loss: 1.4198 - val_acc: 0.5965
Epoch 10/10
5526/5526 [==============================] - 2s 391us/step - loss: 0.0449 - acc: 0.9897 - val_loss: 1.3619 - val_acc: 0.6384
Train on 5526 samples, validate on 11785 samples
Epoch 1/10
5526/5526 [==============================] - 3s 579us/step - loss: 0.6815 - acc: 0.5671 - val_loss: 0.6069 - val_acc: 0.7312
Epoch 2/10
5526/5526 [==============================] - 2s 372us/step - loss: 0.6130 - acc: 0.6826 - val_loss: 0.6413 - val_acc: 0.6594
Epoch 3/10
5526/5526 [==============================] - 2s 379us/step - loss: 0.5116 - acc: 0.7615 - val_loss: 0.6421 - val_acc: 0.6821
Epoch 4/10
5526/5526 [==============================] - 2s 382us/step - loss: 0.3918 - acc: 0.8364 - val_loss: 0.7287 - val_acc: 0.6463
Epoch 5/10
5526/5526 [==============================] - 2s 384us/step - loss: 0.2796 - acc: 0.8969 - val_loss: 0.8899 - val_acc: 0.6034
Epoch 6/10
5526/5526 [==============================] - 2s 385us/step - loss: 0.1897 - acc: 0.9358 - val_loss: 0.9355 - val_acc: 0.6423
Epoch 7/10
5526/5526 [==============================] - 2s 395us/step - loss: 0.1192 - acc: 0.9674 - val_loss: 0.9515 - val_acc: 0.6738
Epoch 8/10
5526/5526 [==============================] - 2s 400us/step - loss: 0.0830 - acc: 0.9761 - val_loss: 1.2283 - val_acc: 0.6207
Epoch 9/10
5526/5526 [==============================] - 2s 396us/step - loss: 0.0549 - acc: 0.9861 - val_loss: 1.1856 - val_acc: 0.6696
Epoch 10/10
5526/5526 [==============================] - 2s 387us/step - loss: 0.0436 - acc: 0.9868 - val_loss: 1.4850 - val_acc: 0.6119
Train on 5526 samples, validate on 11785 samples
Epoch 1/10
5526/5526 [==============================] - 3s 574us/step - loss: 0.6793 - acc: 0.5554 - val_loss: 0.7244 - val_acc: 0.5751
Epoch 2/10
5526/5526 [==============================] - 2s 367us/step - loss: 0.6161 - acc: 0.6880 - val_loss: 0.7274 - val_acc: 0.6141
Epoch 3/10
5526/5526 [==============================] - 2s 384us/step - loss: 0.5303 - acc: 0.7533 - val_loss: 0.8957 - val_acc: 0.5308
Epoch 4/10
5526/5526 [==============================] - 2s 394us/step - loss: 0.4233 - acc: 0.8165 - val_loss: 0.7888 - val_acc: 0.6514
Epoch 5/10
5526/5526 [==============================] - 2s 399us/step - loss: 0.3148 - acc: 0.8740 - val_loss: 1.1240 - val_acc: 0.5683
Epoch 6/10
5526/5526 [==============================] - 2s 395us/step - loss: 0.2197 - acc: 0.9240 - val_loss: 1.4346 - val_acc: 0.5453
Epoch 7/10
5526/5526 [==============================] - 2s 404us/step - loss: 0.1614 - acc: 0.9491 - val_loss: 1.5415 - val_acc: 0.5693
Epoch 8/10
5526/5526 [==============================] - 2s 409us/step - loss: 0.1147 - acc: 0.9678 - val_loss: 1.9303 - val_acc: 0.5285
Epoch 9/10
5526/5526 [==============================] - 2s 393us/step - loss: 0.0841 - acc: 0.9747 - val_loss: 2.1173 - val_acc: 0.5452
Epoch 10/10
5526/5526 [==============================] - 2s 390us/step - loss: 0.0660 - acc: 0.9817 - val_loss: 2.0304 - val_acc: 0.5735
Train on 5528 samples, validate on 11784 samples
Epoch 1/10
5528/5528 [==============================] - 3s 586us/step - loss: 0.6795 - acc: 0.5666 - val_loss: 0.7011 - val_acc: 0.6142
Epoch 2/10
5528/5528 [==============================] - 2s 373us/step - loss: 0.6072 - acc: 0.6943 - val_loss: 0.6921 - val_acc: 0.6529
Epoch 3/10
5528/5528 [==============================] - 2s 375us/step - loss: 0.5200 - acc: 0.7473 - val_loss: 0.8405 - val_acc: 0.5698
Epoch 4/10
5528/5528 [==============================] - 2s 390us/step - loss: 0.4002 - acc: 0.8327 - val_loss: 0.9552 - val_acc: 0.5876
Epoch 5/10
5528/5528 [==============================] - 2s 395us/step - loss: 0.2809 - acc: 0.8900 - val_loss: 1.5983 - val_acc: 0.4430
Epoch 6/10
5528/5528 [==============================] - 2s 392us/step - loss: 0.2051 - acc: 0.9278 - val_loss: 1.2855 - val_acc: 0.5822
Epoch 7/10
5528/5528 [==============================] - 2s 396us/step - loss: 0.1349 - acc: 0.9591 - val_loss: 1.4734 - val_acc: 0.5900
Epoch 8/10
5528/5528 [==============================] - 2s 393us/step - loss: 0.0938 - acc: 0.9714 - val_loss: 1.8799 - val_acc: 0.5565
Epoch 9/10
5528/5528 [==============================] - 2s 395us/step - loss: 0.0700 - acc: 0.9796 - val_loss: 2.1239 - val_acc: 0.5503
Epoch 10/10
5528/5528 [==============================] - 2s 390us/step - loss: 0.0553 - acc: 0.9844 - val_loss: 2.0853 - val_acc: 0.5771
Neural Network(Deep Learning), Under-Sampling

confusion matrix:
[[34268 21203]
 [ 1503  1951]]
sensitivity score:0.5649
specificity score:0.6178
precision score:0.08426
accuracy score:0.6147

5 Model Comparision¶

The classes are highly imbalanced, so we cannot simply count on accuracy to evaluate model performance. We computed four performance scores, including sensitivity, specificity, precision and accuracy. We faced the sensitivity and specificity trade-off when tuning parameters. The goal of the project is to predict patients at high risk of readmission to provide them with special care, so we emphasized the sensitivity score when evaluating model performance.

5.1 Confusion Matrix¶

def draw_conf_mat(conf_mat, title, ax):
    conf_plot = ax.matshow(conf_mat, cmap=plt.cm.Blues)
    class_names = ['Not', 'Readmit']
    ax.set_xticklabels([''] + class_names,fontdict={'fontsize':'20'})
    ax.set_yticklabels([''] + class_names,fontdict={'fontsize':'20'})
    plt.title(title, fontsize='28')
    plt.xlabel('Prediction', fontsize='20')
    plt.ylabel('Truth', fontsize='20')
    num = conf_mat.ravel()
    plt.text(-.15, 0, str(num[0]), fontdict={'color': 'white', 'size':28})
    plt.text(.85, 0, str(num[1]), fontdict={'color': 'black', 'size':28})
    plt.text(-.15, 1, str(num[2]), fontdict={'size':28})
    plt.text(.85, 1, str(num[3]), fontdict={'size':28})

conf_mat_list = [lr_os_conf, lr_us_conf, blr_os_conf,blr_us_conf,dt_conf, rf_conf, nn_conf, nn_us_conf]
clf = ['Logistic Regression_OverSampling', 'Logistic Regression_UnderSampling', 
               'Bagging Logistic Regression_OverSampling', 'Bagging Logistic Regression_UnderSampling', 
               'Decision Tree', 'Random Forest', 'Neural Network', 'Neural Network_UnderSampling']

fig = plt.figure(figsize=(22,22))
for i in range(1, 5):
    ax = fig.add_subplot(int('22{}'.format(i)))
    draw_conf_mat(conf_mat_list[i-1], clf[i-1], ax)
    if i!=1 and i!=3:
        ax.set_ylabel('')

fig = plt.figure(figsize=(20,20))
for i in range(5, 9):
    ax = fig.add_subplot(int('22{}'.format(i-4)))
    draw_conf_mat(conf_mat_list[i-1], clf[i-1], ax)
    if i!=5 and i!=7:
        ax.set_ylabel('')

5.2 Performance Scores¶

clf = ['LR_OS', 'LR_US', 'BagLR_OS', 'BagLR_US', 'DT', 'RF', 'NN', 'NN_US']
sen = [lr_os_sen, lr_us_sen, blr_os_sen, blr_us_sen, dt_sen, rf_sen, nn_sen, nn_us_sen]
spe = [lr_os_spe, lr_us_spe, blr_os_spe, blr_us_spe, dt_spe, rf_spe, nn_spe, nn_us_spe]
prec = [lr_os_prec, lr_us_prec, blr_os_prec, blr_us_prec, dt_prec, rf_prec, nn_prec, nn_us_prec]
acc = [lr_os_acc, lr_us_acc, blr_os_acc, blr_us_acc, dt_acc, rf_acc, nn_acc, nn_us_acc]

perf_dict = {'classifier': clf, 'sensitivity': sen, 'specificity':spe, 'precision': prec, 'accuracy':acc}

draw_df = pd.DataFrame(perf_dict)
draw_melt_df = pd.melt(draw_df, id_vars=['classifier'])

fig = plt.figure(figsize=(30,30))
perf_plt = sns.catplot(x='classifier', y='value',col='variable', col_wrap=2,
                       kind='bar', data=draw_melt_df, palette=sns.color_palette('Paired'))
perf_plt.axes[0].set_title('Sensitivity', fontsize='20')
perf_plt.axes[1].set_title('Specificity', fontsize='20')
perf_plt.axes[2].set_title('Precision',fontsize='20')
perf_plt.axes[3].set_title('Accuracy',fontsize='20')
perf_plt.axes[2].set_xticklabels(labels= clf, rotation='vertical',fontdict={'size':20})
perf_plt.axes[3].set_xticklabels(labels= clf, rotation='vertical',fontdict={'size':20})
perf_plt.axes[2].set_xlabel('')
perf_plt.axes[3].set_xlabel('')
perf_plt.axes[0].set_ylabel('Score', fontsize='20')
perf_plt.axes[2].set_ylabel('Score', fontsize='20')
perf_plt.axes[0].set_yticklabels(labels=['0','0.2','0.4','0.6','0.8','1.0'], fontdict={'size':20})
perf_plt.axes[2].set_yticklabels(labels=['0','0.2','0.4','0.6','0.8','1.0'], fontdict={'size':20})
for index, row in draw_df.iterrows():
    perf_plt.axes[0].text(index-.35, row.sensitivity+.015, round(row.sensitivity,2), fontdict={'size':14})
    perf_plt.axes[1].text(index-.35, row.specificity+.015, round(row.specificity,2), fontdict={'size':14})
    perf_plt.axes[2].text(index-.35, row.precision+.015, round(row.precision,2), fontdict={'size':14})
    perf_plt.axes[3].text(index-.35, row.accuracy+.015, round(row.accuracy,2), fontdict={'size':14})

<Figure size 9000x9000 with 0 Axes>

6 Conclusion¶

Compared to other readmission prediction work, our classifiers perform fairly. One of the challenges in this project is that readmission data is inherently imbalanced. This gives rise to issues regarding designing the model as well as evaluating the model. Class weights can be modified in certain classifiers like random forest and neural network. When it comes to logistic regression, we employed statistic techniques like down sampling and over sampling. Accuracy is obviously inappropriate here. Therefore we also digged into model's sensitivity, specificity, precision and confusion matrix. Ensemble methods such as random forest and bagging logistic regression turn out to provide better performance than a single classifier. In terms of neural networks, we started from empirically good networks and modify hyper parameters to try to achieve better performance.

	type	name	tbl_name	rootpage	sql
0	table	CHARTEVENTS	CHARTEVENTS	2	CREATE TABLE CHARTEVENTS(\n "ROW_ID" TEXT,\n ...
1	table	PROCEDUREEVENTS_MV	PROCEDUREEVENTS_MV	9120960	CREATE TABLE "PROCEDUREEVENTS_MV" (\n"index" I...
2	table	CALLOUT	CALLOUT	9132354	CREATE TABLE "CALLOUT" (\n"index" INTEGER,\n ...
3	table	D_CPT	D_CPT	9133903	CREATE TABLE "D_CPT" (\n"index" INTEGER,\n "R...
4	table	D_ITEMS	D_ITEMS	9133909	CREATE TABLE "D_ITEMS" (\n"index" INTEGER,\n ...
5	table	CAREGIVERS	CAREGIVERS	9134171	CREATE TABLE "CAREGIVERS" (\n"index" INTEGER,\...
6	table	MICROBIOLOGYEVENTS	MICROBIOLOGYEVENTS	9134246	CREATE TABLE "MICROBIOLOGYEVENTS" (\n"index" I...
7	table	LABEVENTS	LABEVENTS	9153054	CREATE TABLE "LABEVENTS" (\n"index" INTEGER,\n...
8	table	INPUTEVENTS_CV	INPUTEVENTS_CV	9708019	CREATE TABLE "INPUTEVENTS_CV" (\n"index" INTEG...
9	table	ADMISSIONS	ADMISSIONS	10291267	CREATE TABLE "ADMISSIONS" (\n"index" INTEGER,\...
10	table	D_LABITEMS	D_LABITEMS	10294375	CREATE TABLE "D_LABITEMS" (\n"index" INTEGER,\...
11	table	DATETIMEEVENTS	DATETIMEEVENTS	10294390	CREATE TABLE "DATETIMEEVENTS" (\n"index" INTEG...
12	table	PRESCRIPTIONS	PRESCRIPTIONS	10428841	CREATE TABLE "PRESCRIPTIONS" (\n"index" INTEGE...
13	table	PROCEDURES_ICD	PROCEDURES_ICD	10605718	CREATE TABLE "PROCEDURES_ICD" (\n"index" INTEG...
14	table	NOTEEVENTS	NOTEEVENTS	10608088	CREATE TABLE "NOTEEVENTS" (\n"index" INTEGER,\...
15	table	TRANSFERS	TRANSFERS	11772552	CREATE TABLE "TRANSFERS" (\n"index" INTEGER,\n...
16	table	DIAGNOSES_ICD	DIAGNOSES_ICD	11779463	CREATE TABLE "DIAGNOSES_ICD" (\n"index" INTEGE...
17	table	SERVICES	SERVICES	11786385	CREATE TABLE "SERVICES" (\n"index" INTEGER,\n ...
18	table	DRGCODES	DRGCODES	11787476	CREATE TABLE "DRGCODES" (\n"index" INTEGER,\n ...
19	table	OUTPUTEVENTS	OUTPUTEVENTS	11790336	CREATE TABLE "OUTPUTEVENTS" (\n"index" INTEGER...
20	table	PATIENTS	PATIENTS	11893144	CREATE TABLE "PATIENTS" (\n"index" INTEGER,\n ...
21	table	D_ICD_DIAGNOSES	D_ICD_DIAGNOSES	11893958	CREATE TABLE "D_ICD_DIAGNOSES" (\n"index" INTE...
22	table	ICUSTAYS	ICUSTAYS	11894350	CREATE TABLE "ICUSTAYS" (\n"index" INTEGER,\n ...
23	table	INPUTEVENTS_MV	INPUTEVENTS_MV	11896023	CREATE TABLE "INPUTEVENTS_MV" (\n"index" INTEG...
24	table	D_ICD_PROCEDURES	D_ICD_PROCEDURES	12123226	CREATE TABLE "D_ICD_PROCEDURES" (\n"index" INT...
25	table	CPTEVENTS	CPTEVENTS	12123316	CREATE TABLE "CPTEVENTS" (\n"index" INTEGER,\n...

	ROW_ID	SUBJECT_ID	HADM_ID	ICUSTAY_ID	ITEMID	CHARTTIME	STORETIME	CGID	VALUE	VALUENUM	VALUEUOM
0	788	36	165660	241249	223834	2134-05-12 12:00:00	2134-05-12 13:56:00	17525	15	15	L/min
1	789	36	165660	241249	223835	2134-05-12 12:00:00	2134-05-12 13:56:00	17525	100	100
2	790	36	165660	241249	224328	2134-05-12 12:00:00	2134-05-12 12:18:00	20823	.37	.37
3	791	36	165660	241249	224329	2134-05-12 12:00:00	2134-05-12 12:19:00	20823	6	6	min
4	792	36	165660	241249	224330	2134-05-12 12:00:00	2134-05-12 12:19:00	20823	2.5	2.5

	index	ROW_ID	SUBJECT_ID	HADM_ID	ICUSTAY_ID	STARTTIME	ENDTIME	ITEMID	VALUE	VALUEUOM	LOCATION	LOCATIONCATEGORY	STORETIME	CGID	ORDERID	LINKORDERID	ORDERCATEGORYNAME	SECONDARYORDERCATEGORYNAME	ORDERCATEGORYDESCRIPTION	ISOPENBAG	STATUSDESCRIPTION	COMMENTS_EDITEDBY	COMMENTS_CANCELEDBY	COMMENTS_DATE
0	0	379	29070	115071	232563.0	2145-03-12 23:04:00	2145-03-12 23:05:00	225401	1.0	None	None	None	2145-03-12 23:04:00	20855	5092605	5092605	Procedures	None	Electrolytes	0	FinishedRunning	None	None	None
1	1	380	29070	115071	232563.0	2145-03-12 23:04:00	2145-03-12 23:05:00	225454	1.0	None	None	None	2145-03-12 23:04:00	20855	2538599	2538599	Procedures	None	Electrolytes	0	FinishedRunning	None	None	None
2	2	381	29070	115071	232563.0	2145-03-12 23:05:00	2145-03-18 20:01:00	225792	8456.0	hour	None	None	2145-03-18 20:01:39	20889	2383835	2383835	Ventilation	None	Task	1	FinishedRunning	None	None	None
3	3	382	29070	115071	232563.0	2145-03-12 23:36:00	2145-03-12 23:37:00	225402	1.0	None	None	None	2145-03-12 23:36:00	20855	9355062	9355062	Procedures	None	Electrolytes	0	FinishedRunning	None	None	None
4	4	383	29070	115071	232563.0	2145-03-13 01:27:00	2145-03-16 16:00:00	224560	5193.0	min	Right IJ	Invasive Venous	2145-03-16 17:57:00	20855	466169	466169	Invasive Lines	None	Task	1	FinishedRunning	None	None	None

	index	ROW_ID	SUBJECT_ID	HADM_ID	SUBMIT_WARDID	SUBMIT_CAREUNIT	CURR_WARDID	CURR_CAREUNIT	CALLOUT_WARDID	CALLOUT_SERVICE	REQUEST_TELE	REQUEST_RESP	CALLOUT_STATUS	CALLOUT_OUTCOME	DISCHARGE_WARDID	ACKNOWLEDGE_STATUS	CREATETIME	UPDATETIME	ACKNOWLEDGETIME	OUTCOMETIME	FIRSTRESERVATIONTIME	CURRENTRESERVATIONTIME
0	0	402	854	175684	52.0	None	29.0	MICU	1	MED	0	0	Inactive	Discharged	29.0	Acknowledged	2146-10-05 13:16:55	2146-10-05 13:16:55	2146-10-05 13:24:00	2146-10-05 18:55:22	2146-10-05 15:27:44	None
1	1	403	864	138624	15.0	None	55.0	CSRU	55	CSURG	0	0	Inactive	Discharged	55.0	Acknowledged	2114-11-28 08:31:39	2114-11-28 09:42:08	2114-11-28 09:43:08	2114-11-28 12:10:02	None	None
2	2	404	864	138624	12.0	None	55.0	CSRU	55	CSURG	1	0	Inactive	Discharged	55.0	Acknowledged	2114-11-30 10:24:25	2114-12-01 09:06:18	2114-12-01 12:26:05	2114-12-01 21:55:05	None	None
3	3	405	867	184298	7.0	None	17.0	CCU	17	CCU	1	1	Inactive	Discharged	17.0	Acknowledged	2136-12-29 08:45:42	2136-12-29 10:17:16	2136-12-29 10:33:51	2136-12-29 18:10:02	None	None
4	4	157	306	167129	57.0	None	3.0	SICU	44	NSURG	1	0	Inactive	Discharged	3.0	Acknowledged	2199-09-18 11:47:47	2199-09-18 11:47:47	2199-09-18 11:58:33	2199-09-18 15:10:02	None	None

	index	ROW_ID	CATEGORY	SECTIONRANGE	SECTIONHEADER	SUBSECTIONRANGE	SUBSECTIONHEADER	CODESUFFIX	MINCODEINSUBSECTION	MAXCODEINSUBSECTION
0	0	1	1	99201-99499	Evaluation and management	99201-99216	Office/other outpatient services	None	99201	99216
1	1	2	1	99201-99499	Evaluation and management	99217-99220	Hospital observation services	None	99217	99220
2	2	3	1	99201-99499	Evaluation and management	99221-99239	Hospital inpatient services	None	99221	99239
3	3	4	1	99201-99499	Evaluation and management	99241-99255	Consultations	None	99241	99255
4	4	5	1	99201-99499	Evaluation and management	99261-99263	Follow-up inpatient consultations (deleted codes)	None	99261	99263

	index	ROW_ID	ITEMID	LABEL	ABBREVIATION	DBSOURCE	LINKSTO	CATEGORY	UNITNAME	PARAM_TYPE	CONCEPTID
0	0	457	497	Patient controlled analgesia (PCA) [Inject]	None	carevue	chartevents	None	None	None	None
1	1	458	498	PCA Lockout (Min)	None	carevue	chartevents	None	None	None	None
2	2	459	499	PCA Medication	None	carevue	chartevents	None	None	None	None
3	3	460	500	PCA Total Dose	None	carevue	chartevents	None	None	None	None
4	4	461	501	PCV Exh Vt (Obser)	None	carevue	chartevents	None	None	None	None

	index	ROW_ID	CGID	LABEL	DESCRIPTION
0	0	2228	16174	RO	Read Only
1	1	2229	16175	RO	Read Only
2	2	2230	16176	Res	Resident/Fellow/PA/NP
3	3	2231	16177	RO	Read Only
4	4	2232	16178	RT	Respiratory

	index	ROW_ID	SUBJECT_ID	HADM_ID	CHARTDATE	CHARTTIME	SPEC_ITEMID	SPEC_TYPE_DESC	ORG_ITEMID	ORG_NAME	ISOLATE_NUM	AB_ITEMID	AB_NAME	DILUTION_TEXT	DILUTION_COMPARISON	DILUTION_VALUE	INTERPRETATION
0	0	744	96	170324	2156-04-13 00:00:00	2156-04-13 14:18:00	70021.0	BRONCHOALVEOLAR LAVAGE	80026.0	PSEUDOMONAS AERUGINOSA	1.0	None	None	None	None	None	None
1	1	745	96	170324	2156-04-20 00:00:00	2156-04-20 13:10:00	70062.0	SPUTUM	NaN	None	NaN	None	None	None	None	None	None
2	2	746	96	170324	2156-04-20 00:00:00	2156-04-20 16:00:00	70012.0	BLOOD CULTURE	NaN	None	NaN	None	None	None	None	None	None
3	3	747	96	170324	2156-04-20 00:00:00	None	70012.0	BLOOD CULTURE	NaN	None	NaN	None	None	None	None	None	None
4	4	748	96	170324	2156-04-20 00:00:00	None	70079.0	URINE	NaN	None	NaN	None	None	None	None	None	None

	index	ROW_ID	SUBJECT_ID	HADM_ID	ITEMID	CHARTTIME	VALUE	VALUENUM	VALUEUOM	FLAG
0	0	281	3	None	50820	2101-10-12 16:07:00	7.39	7.39	units	None
1	1	282	3	None	50800	2101-10-12 18:17:00	ART	NaN	None	None
2	2	283	3	None	50802	2101-10-12 18:17:00	-1	-1.00	mEq/L	None
3	3	284	3	None	50804	2101-10-12 18:17:00	22	22.00	mEq/L	None
4	4	285	3	None	50808	2101-10-12 18:17:00	0.93	0.93	mmol/L	abnormal

	index	ROW_ID	SUBJECT_ID	HADM_ID	ICUSTAY_ID	CHARTTIME	ITEMID	AMOUNT	AMOUNTUOM	RATE	RATEUOM	STORETIME	CGID	ORDERID	LINKORDERID	STOPPED	NEWBOTTLE	ORIGINALAMOUNT	ORIGINALAMOUNTUOM	ORIGINALROUTE	ORIGINALRATE	ORIGINALRATEUOM	ORIGINALSITE
0	0	592	24457	184834.0	205776.0	2193-09-11 09:00:00	30056	100.0	ml	None	None	2193-09-11 11:12:00	14990.0	756654	9359133	None	None	None	ml	Oral	None	None	None
1	1	593	24457	184834.0	205776.0	2193-09-11 12:00:00	30056	200.0	ml	None	None	2193-09-11 12:05:00	14990.0	3564075	9359133	None	None	None	ml	Oral	None	None	None
2	2	594	24457	184834.0	205776.0	2193-09-11 16:00:00	30056	160.0	ml	None	None	2193-09-11 16:52:00	14990.0	422646	9359133	None	None	None	ml	Oral	None	None	None
3	3	595	24457	184834.0	205776.0	2193-09-11 19:00:00	30056	240.0	ml	None	None	2193-09-11 19:06:00	14990.0	5137889	9359133	None	None	None	ml	Oral	None	None	None
4	4	596	24457	184834.0	205776.0	2193-09-11 21:00:00	30056	50.0	ml	None	None	2193-09-11 21:19:00	15029.0	8343792	9359133	None	None	None	ml	Oral	None	None	None

	index	ROW_ID	SUBJECT_ID	HADM_ID	ADMITTIME	DISCHTIME	DEATHTIME	ADMISSION_TYPE	ADMISSION_LOCATION	DISCHARGE_LOCATION	INSURANCE	LANGUAGE	RELIGION	MARITAL_STATUS	ETHNICITY	EDREGTIME	EDOUTTIME	DIAGNOSIS	HAS_CHARTEVENTS_DATA
0	0	21	22	165315	2196-04-09 12:26:00	2196-04-10 15:54:00	None	EMERGENCY	EMERGENCY ROOM ADMIT	DISC-TRAN CANCER/CHLDRN H	Private	None	UNOBTAINABLE	MARRIED	WHITE	2196-04-09 10:06:00	2196-04-09 13:24:00	BENZODIAZEPINE OVERDOSE	1
1	1	22	23	152223	2153-09-03 07:15:00	2153-09-08 19:10:00	None	ELECTIVE	PHYS REFERRAL/NORMAL DELI	HOME HEALTH CARE	Medicare	None	CATHOLIC	MARRIED	WHITE	None	None	CORONARY ARTERY DISEASE\CORONARY ARTERY BYPASS...	1
2	2	23	23	124321	2157-10-18 19:34:00	2157-10-25 14:00:00	None	EMERGENCY	TRANSFER FROM HOSP/EXTRAM	HOME HEALTH CARE	Medicare	ENGL	CATHOLIC	MARRIED	WHITE	None	None	BRAIN MASS	1
3	3	24	24	161859	2139-06-06 16:14:00	2139-06-09 12:48:00	None	EMERGENCY	TRANSFER FROM HOSP/EXTRAM	HOME	Private	None	PROTESTANT QUAKER	SINGLE	WHITE	None	None	INTERIOR MYOCARDIAL INFARCTION	1
4	4	25	25	129635	2160-11-02 02:06:00	2160-11-05 14:55:00	None	EMERGENCY	EMERGENCY ROOM ADMIT	HOME	Private	None	UNOBTAINABLE	MARRIED	WHITE	2160-11-02 01:01:00	2160-11-02 04:27:00	ACUTE CORONARY SYNDROME	1

	index	ROW_ID	ITEMID	LABEL	FLUID	CATEGORY	LOINC_CODE
0	0	546	51346	Blasts	Cerebrospinal Fluid (CSF)	Hematology	26447-3
1	1	547	51347	Eosinophils	Cerebrospinal Fluid (CSF)	Hematology	26451-5
2	2	548	51348	Hematocrit, CSF	Cerebrospinal Fluid (CSF)	Hematology	30398-2
3	3	549	51349	Hypersegmented Neutrophils	Cerebrospinal Fluid (CSF)	Hematology	26506-6
4	4	550	51350	Immunophenotyping	Cerebrospinal Fluid (CSF)	Hematology	None

	index	ROW_ID	SUBJECT_ID	HADM_ID	ICUSTAY_ID	ITEMID	CHARTTIME	STORETIME	CGID	VALUE	VALUEUOM	WARNING	ERROR	RESULTSTATUS	STOPPED
0	0	711	7657	121183.0	297945.0	3411	2172-03-14 11:00:00	2172-03-14 11:52:00	16446	None	Date	None	None	None	NotStopd
1	1	712	7657	121183.0	297945.0	3411	2172-03-14 13:00:00	2172-03-14 12:36:00	16446	None	Date	None	None	None	NotStopd
2	2	713	7657	121183.0	297945.0	3411	2172-03-14 15:00:00	2172-03-14 15:10:00	14957	None	Date	None	None	None	NotStopd
3	3	714	7657	121183.0	297945.0	3411	2172-03-14 17:00:00	2172-03-14 17:01:00	16446	None	Date	None	None	None	NotStopd
4	4	715	7657	121183.0	297945.0	3411	2172-03-14 19:00:00	2172-03-14 19:29:00	14815	None	Date	None	None	None	NotStopd

	index	ROW_ID	SUBJECT_ID	HADM_ID	ICUSTAY_ID	STARTDATE	ENDDATE	DRUG_TYPE	DRUG	DRUG_NAME_POE	DRUG_NAME_GENERIC	FORMULARY_DRUG_CD	GSN	NDC	PROD_STRENGTH	DOSE_VAL_RX	DOSE_UNIT_RX	FORM_VAL_DISP	FORM_UNIT_DISP	ROUTE
0	0	2214776	6	107064	None	2175-06-11 00:00:00	2175-06-12 00:00:00	MAIN	Tacrolimus	Tacrolimus	Tacrolimus	TACR1	021796	469061711.0	1mg Capsule	2	mg	2	CAP	PO
1	1	2214775	6	107064	None	2175-06-11 00:00:00	2175-06-12 00:00:00	MAIN	Warfarin	Warfarin	Warfarin	WARF5	006562	56017275.0	5mg Tablet	5	mg	1	TAB	PO
2	2	2215524	6	107064	None	2175-06-11 00:00:00	2175-06-12 00:00:00	MAIN	Heparin Sodium	None	None	HEPAPREMIX	006522	338055002.0	25,000 unit Premix Bag	25,000	UNIT	1	BAG	IV
3	3	2216265	6	107064	None	2175-06-11 00:00:00	2175-06-12 00:00:00	BASE	D5W	None	None	HEPBASE	None	0.0	HEPARIN BASE	250	ml	250	ml	IV
4	4	2214773	6	107064	None	2175-06-11 00:00:00	2175-06-12 00:00:00	MAIN	Furosemide	Furosemide	Furosemide	FURO20	008208	54829725.0	20mg Tablet	20	mg	1	TAB	PO

	index	ROW_ID	SUBJECT_ID	HADM_ID	SEQ_NUM	ICD9_CODE
0	0	944	62641	154460	3	3404
1	1	945	2592	130856	1	9671
2	2	946	2592	130856	2	3893
3	3	947	55357	119355	1	9672
4	4	948	55357	119355	2	331

	index	ROW_ID	SUBJECT_ID	HADM_ID	CHARTDATE	CHARTTIME	STORETIME	CATEGORY	DESCRIPTION	CGID	ISERROR	TEXT
0	0	174	22532	167853.0	2151-08-04	None	None	Discharge summary	Report	None	None	Admission Date: [2151-7-16] Dischar...
1	1	175	13702	107527.0	2118-06-14	None	None	Discharge summary	Report	None	None	Admission Date: [2118-6-2] Discharg...
2	2	176	13702	167118.0	2119-05-25	None	None	Discharge summary	Report	None	None	Admission Date: [2119-5-4] D...
3	3	177	13702	196489.0	2124-08-18	None	None	Discharge summary	Report	None	None	Admission Date: [2124-7-21] ...
4	4	178	26880	135453.0	2162-03-25	None	None	Discharge summary	Report	None	None	Admission Date: [2162-3-3] D...

	index	ROW_ID	SUBJECT_ID	HADM_ID	ICUSTAY_ID	DBSOURCE	EVENTTYPE	PREV_CAREUNIT	CURR_CAREUNIT	PREV_WARDID	CURR_WARDID	INTIME	OUTTIME	LOS
0	0	657	111	192123	254245.0	carevue	transfer	CCU	MICU	7.0	23.0	2142-04-29 15:27:11	2142-05-04 20:38:33	125.19
1	1	658	111	192123	NaN	carevue	transfer	MICU	None	23.0	45.0	2142-05-04 20:38:33	2142-05-05 11:46:32	15.13
2	2	659	111	192123	NaN	carevue	discharge	None	None	45.0	NaN	2142-05-05 11:46:32	None	NaN
3	3	660	111	155897	249202.0	metavision	admit	None	MICU	NaN	52.0	2144-07-01 04:13:59	2144-07-01 05:19:39	1.09
4	4	661	111	155897	NaN	metavision	transfer	MICU	None	52.0	32.0	2144-07-01 05:19:39	2144-07-01 06:28:29	1.15

	index	ROW_ID	SUBJECT_ID	HADM_ID	SEQ_NUM	ICD9_CODE
0	0	1297	109	172335	1.0	40301
1	1	1298	109	172335	2.0	486
2	2	1299	109	172335	3.0	58281
3	3	1300	109	172335	4.0	5855
4	4	1301	109	172335	5.0	4254

	index	ROW_ID	SUBJECT_ID	HADM_ID	TRANSFERTIME	PREV_SERVICE	CURR_SERVICE
0	0	758	471	135879	2122-07-22 14:07:27	TSURG	MED
1	1	759	471	135879	2122-07-26 18:31:49	MED	TSURG
2	2	760	472	173064	2172-09-28 19:22:15	None	CMED
3	3	761	473	129194	2201-01-09 20:16:45	None	NB
4	4	762	474	194246	2181-03-23 08:24:41	None	NB

	index	ROW_ID	SUBJECT_ID	HADM_ID	DRG_TYPE	DRG_CODE	DESCRIPTION	DRG_SEVERITY	DRG_MORTALITY
0	0	342	2491	144486	HCFA	28	TRAUMATIC STUPOR & COMA, COMA <1 HR AGE >17 WI...	None	None
1	1	343	24958	162910	HCFA	110	MAJOR CARDIOVASCULAR PROCEDURES WITH COMPLICAT...	None	None
2	2	344	18325	153751	HCFA	390	NEONATE WITH OTHER SIGNIFICANT PROBLEMS	None	None
3	3	345	17887	182692	HCFA	14	SPECIFIC CEREBROVASCULAR DISORDERS EXCEPT TRAN...	None	None
4	4	346	11113	157980	HCFA	390	NEONATE WITH OTHER SIGNIFICANT PROBLEMS	None	None

	index	ROW_ID	SUBJECT_ID	HADM_ID	ICUSTAY_ID	CHARTTIME	ITEMID	VALUE	VALUEUOM	STORETIME	CGID	STOPPED	NEWBOTTLE	ISERROR
0	0	344	21219	177991.0	225765.0	2142-09-08 10:00:00	40055	200.0	ml	2142-09-08 12:08:00	17269	None	None	None
1	1	345	21219	177991.0	225765.0	2142-09-08 12:00:00	40055	200.0	ml	2142-09-08 12:08:00	17269	None	None	None
2	2	346	21219	177991.0	225765.0	2142-09-08 13:00:00	40055	120.0	ml	2142-09-08 13:39:00	17269	None	None	None
3	3	347	21219	177991.0	225765.0	2142-09-08 14:00:00	40055	100.0	ml	2142-09-08 16:17:00	17269	None	None	None
4	4	348	21219	177991.0	225765.0	2142-09-08 16:00:00	40055	200.0	ml	2142-09-08 16:17:00	17269	None	None	None

	index	ROW_ID	SUBJECT_ID	GENDER	DOB	DOD	DOD_HOSP	DOD_SSN	EXPIRE_FLAG
0	0	234	249	F	2075-03-13 00:00:00	None	None	None	0
1	1	235	250	F	2164-12-27 00:00:00	2188-11-22 00:00:00	2188-11-22 00:00:00	None	1
2	2	236	251	M	2090-03-15 00:00:00	None	None	None	0
3	3	237	252	M	2078-03-06 00:00:00	None	None	None	0
4	4	238	253	F	2089-11-26 00:00:00	None	None	None	0

	index	ROW_ID	ICD9_CODE	SHORT_TITLE	LONG_TITLE
0	0	174	01166	TB pneumonia-oth test	Tuberculous pneumonia [any form], tubercle bac...
1	1	175	01170	TB pneumothorax-unspec	Tuberculous pneumothorax, unspecified
2	2	176	01171	TB pneumothorax-no exam	Tuberculous pneumothorax, bacteriological or h...
3	3	177	01172	TB pneumothorx-exam unkn	Tuberculous pneumothorax, bacteriological or h...
4	4	178	01173	TB pneumothorax-micro dx	Tuberculous pneumothorax, tubercle bacilli fou...

	index	ROW_ID	SUBJECT_ID	HADM_ID	ICUSTAY_ID	DBSOURCE	FIRST_CAREUNIT	LAST_CAREUNIT	FIRST_WARDID	LAST_WARDID	INTIME	OUTTIME	LOS
0	0	365	268	110404	280836	carevue	MICU	MICU	52	52	2198-02-14 23:27:38	2198-02-18 05:26:11	3.2490
1	1	366	269	106296	206613	carevue	MICU	MICU	52	52	2170-11-05 11:05:29	2170-11-08 17:46:57	3.2788
2	2	367	270	188028	220345	carevue	CCU	CCU	57	57	2128-06-24 15:05:20	2128-06-27 12:32:29	2.8939
3	3	368	271	173727	249196	carevue	MICU	SICU	52	23	2120-08-07 23:12:42	2120-08-10 00:39:04	2.0600
4	4	369	272	164716	210407	carevue	CCU	CCU	57	57	2186-12-25 21:08:04	2186-12-27 12:01:13	1.6202

	index	ROW_ID	SUBJECT_ID	HADM_ID	ICUSTAY_ID	STARTTIME	ENDTIME	ITEMID	AMOUNT	AMOUNTUOM	RATE	RATEUOM	STORETIME	CGID	ORDERID	LINKORDERID	ORDERCATEGORYNAME	SECONDARYORDERCATEGORYNAME	ORDERCOMPONENTTYPEDESCRIPTION	ORDERCATEGORYDESCRIPTION	PATIENTWEIGHT	TOTALAMOUNT	TOTALAMOUNTUOM	CANCELREASON	STATUSDESCRIPTION	COMMENTS_EDITEDBY	COMMENTS_CANCELEDBY	COMMENTS_DATE	ORIGINALAMOUNT	ORIGINALRATE
0	0	241	27063	139787	223259.0	2133-02-05 06:29:00	2133-02-05 08:45:00	225166	6.774532	mEq	NaN	None	2133-02-05 06:27:00	16009	4892074	4892074	02-Fluids (Crystalloids)	Additive (Crystalloid)	Additives ...	Continuous IV	83.2	100.0	ml	1	Rewritten	None	RN	2133-02-05 12:52:00	10.000000	0.050000
1	1	242	27063	139787	223259.0	2133-02-05 05:34:00	2133-02-05 06:30:00	225944	28.132997	ml	30.142497	mL/hour	2133-02-05 12:57:00	20053	5211428	1576890	02-Fluids (Crystalloids)	Additive (Crystalloid)	Main order parameter	Continuous IV	83.2	100.0	ml	0	FinishedRunning	None	None	None	28.132998	30.255817
2	2	243	27063	139787	223259.0	2133-02-05 05:34:00	2133-02-05 06:30:00	225166	2.813300	mEq	NaN	None	2133-02-05 12:57:00	20053	5211428	1576890	02-Fluids (Crystalloids)	Additive (Crystalloid)	Additives ...	Continuous IV	83.2	100.0	ml	0	FinishedRunning	None	None	None	2.813300	0.050426
3	3	244	27063	139787	223259.0	2133-02-03 12:00:00	2133-02-03 12:01:00	225893	1.000000	dose	NaN	None	2133-02-03 13:21:00	20030	5213971	5213971	08-Antibiotics (IV)	02-Fluids (Crystalloids)	Main order parameter	Drug Push	83.2	100.0	ml	2	Rewritten	RN	None	2133-02-03 17:06:00	1.000000	1.000000
4	4	245	27063	139787	223259.0	2133-02-03 12:00:00	2133-02-03 12:01:00	220949	100.000000	ml	NaN	None	2133-02-03 13:21:00	20030	5213971	5213971	08-Antibiotics (IV)	02-Fluids (Crystalloids)	Mixed solution	Drug Push	83.2	100.0	ml	2	Rewritten	RN	None	2133-02-03 17:06:00	100.000000	0.000000

	index	ROW_ID	ICD9_CODE	SHORT_TITLE	LONG_TITLE
0	0	264	851	Canthotomy	Canthotomy
1	1	265	852	Blepharorrhaphy	Blepharorrhaphy
2	2	266	859	Adjust lid position NEC	Other adjustment of lid position
3	3	267	861	Lid reconst w skin graft	Reconstruction of eyelid with skin flap or graft
4	4	268	862	Lid reconst w muc graft	Reconstruction of eyelid with mucous membrane ...

	index	ROW_ID	SUBJECT_ID	HADM_ID	COSTCENTER	CHARTDATE	CPT_CD	CPT_NUMBER	CPT_SUFFIX	TICKET_ID_SEQ	SECTIONHEADER	SUBSECTIONHEADER	DESCRIPTION
0	0	317	11743	129545	ICU	None	99232	99232.0	None	6.0	Evaluation and management	Hospital inpatient services	None
1	1	318	11743	129545	ICU	None	99232	99232.0	None	7.0	Evaluation and management	Hospital inpatient services	None
2	2	319	11743	129545	ICU	None	99232	99232.0	None	8.0	Evaluation and management	Hospital inpatient services	None
3	3	320	11743	129545	ICU	None	99232	99232.0	None	9.0	Evaluation and management	Hospital inpatient services	None
4	4	321	6185	183725	ICU	None	99223	99223.0	None	1.0	Evaluation and management	Hospital inpatient services	None

	SUBJECT_ID	HADM_ID	ADMITTIME	DISCHTIME	ADMITTIME_RANK	SUBJECT_ID	MAX_ADMITTIME_RANK	NEXT_ADMITTIME	30_READMIT	INTERVAL
0	2	163353	2138-07-17 19:04:00	2138-07-21 15:48:00	1	2	1	2101-10-20 19:08:00	0	NaN
1	3	145834	2101-10-20 19:08:00	2101-10-31 13:58:00	1	3	1	2191-03-16 00:28:00	0	NaN
2	4	185777	2191-03-16 00:28:00	2191-03-23 18:41:00	1	4	1	2103-02-02 04:31:00	0	NaN
3	5	178980	2103-02-02 04:31:00	2103-02-04 12:15:00	1	5	1	2175-05-30 07:15:00	0	NaN
4	6	107064	2175-05-30 07:15:00	2175-06-15 16:00:00	1	6	1	2121-05-23 15:05:00	0	NaN
5	7	118037	2121-05-23 15:05:00	2121-05-27 11:57:00	1	7	1	2117-11-20 10:22:00	0	NaN
6	8	159514	2117-11-20 10:22:00	2117-11-24 14:20:00	1	8	1	2149-11-09 13:06:00	0	NaN
7	9	150750	2149-11-09 13:06:00	2149-11-14 10:15:00	1	9	1	2103-06-28 11:36:00	0	NaN
8	10	184167	2103-06-28 11:36:00	2103-07-06 12:10:00	1	10	1	2178-04-16 06:18:00	0	NaN
9	11	194540	2178-04-16 06:18:00	2178-05-11 19:00:00	1	11	1	2104-08-07 10:15:00	0	NaN
10	12	112213	2104-08-07 10:15:00	2104-08-20 02:57:00	1	12	1	2167-01-08 18:43:00	0	NaN
11	13	143045	2167-01-08 18:43:00	2167-01-15 15:15:00	1	13	1	2178-02-03 06:35:00	0	NaN
12	16	103251	2178-02-03 06:35:00	2178-02-05 10:51:00	1	16	1	2134-12-27 07:15:00	0	NaN
13	17	194023	2134-12-27 07:15:00	2134-12-31 16:05:00	1	17	2	2135-05-09 14:11:00	0	128.0
14	17	161087	2135-05-09 14:11:00	2135-05-13 14:40:00	2	17	2	2167-10-02 11:18:00	0	NaN
15	18	188822	2167-10-02 11:18:00	2167-10-04 16:15:00	1	18	1	2108-08-05 16:25:00	0	NaN
16	19	109235	2108-08-05 16:25:00	2108-08-11 11:29:00	1	19	1	2183-04-28 09:45:00	0	NaN
17	20	157681	2183-04-28 09:45:00	2183-05-03 14:45:00	1	20	1	2134-09-11 12:17:00	0	NaN
18	21	109451	2134-09-11 12:17:00	2134-09-24 16:15:00	1	21	2	2135-01-30 20:50:00	0	128.0
19	21	111970	2135-01-30 20:50:00	2135-02-08 02:08:00	2	21	2	2196-04-09 12:26:00	0	NaN
20	22	165315	2196-04-09 12:26:00	2196-04-10 15:54:00	1	22	1	2153-09-03 07:15:00	0	NaN
21	23	152223	2153-09-03 07:15:00	2153-09-08 19:10:00	1	23	2	2157-10-18 19:34:00	0	1501.0
22	23	124321	2157-10-18 19:34:00	2157-10-25 14:00:00	2	23	2	2139-06-06 16:14:00	0	NaN
23	24	161859	2139-06-06 16:14:00	2139-06-09 12:48:00	1	24	1	2160-11-02 02:06:00	0	NaN
24	25	129635	2160-11-02 02:06:00	2160-11-05 14:55:00	1	25	1	2126-05-06 15:16:00	0	NaN
25	26	197661	2126-05-06 15:16:00	2126-05-13 15:00:00	1	26	1	2191-11-30 22:16:00	0	NaN
26	27	134931	2191-11-30 22:16:00	2191-12-03 14:45:00	1	27	1	2177-09-01 07:15:00	0	NaN
27	28	162569	2177-09-01 07:15:00	2177-09-06 16:00:00	1	28	1	2172-10-14 14:17:00	0	NaN
28	30	104557	2172-10-14 14:17:00	2172-10-19 14:37:00	1	30	1	2108-08-22 23:27:00	0	NaN
29	31	128652	2108-08-22 23:27:00	2108-08-30 15:00:00	1	31	1	2170-04-04 08:00:00	0	NaN
...	...	...	...	...	...	...	...	...	...	...
58946	99901	131711	2173-01-06 15:51:00	2173-01-19 17:32:00	1	99901	1	2146-01-06 18:17:00	0	NaN
58947	99903	135939	2146-01-06 18:17:00	2146-01-10 18:30:00	1	99903	1	2153-04-13 16:05:00	0	NaN
58948	99912	189380	2153-04-13 16:05:00	2153-04-27 13:40:00	1	99912	1	2141-05-10 20:41:00	0	NaN
58949	99913	175989	2141-05-10 20:41:00	2141-05-18 17:54:00	1	99913	1	2107-04-07 07:15:00	0	NaN
58950	99922	123563	2107-04-07 07:15:00	2107-04-11 17:16:00	1	99922	1	2201-02-23 20:42:00	0	NaN
58951	99923	164914	2201-02-23 20:42:00	2201-03-03 16:00:00	1	99923	2	2201-05-15 13:12:00	0	72.0
58952	99923	192053	2201-05-15 13:12:00	2201-05-25 14:51:00	2	99923	2	2181-11-16 16:20:00	0	NaN
58953	99928	104802	2181-11-16 16:20:00	2181-11-18 14:13:00	1	99928	1	2110-02-28 03:47:00	0	NaN
58954	99934	176121	2110-02-28 03:47:00	2110-03-06 14:31:00	1	99934	1	2139-12-15 06:59:00	0	NaN
58955	99935	174624	2139-12-15 06:59:00	2139-12-15 12:00:00	1	99935	1	2182-10-10 13:57:00	0	NaN
58956	99936	107913	2182-10-10 13:57:00	2182-10-18 13:45:00	1	99936	1	2128-05-11 15:16:00	0	NaN
58957	99937	129380	2128-05-11 15:16:00	2128-05-20 13:35:00	1	99937	1	2200-10-24 17:53:00	0	NaN
58958	99938	103815	2200-10-24 17:53:00	2200-11-11 11:53:00	1	99938	1	2110-03-22 04:03:00	0	NaN
58959	99939	159023	2110-03-22 04:03:00	2110-04-02 15:02:00	1	99939	1	2161-02-16 20:14:00	0	NaN
58960	99944	185654	2161-02-16 20:14:00	2161-02-21 16:45:00	1	99944	1	2161-07-14 20:29:00	0	NaN
58961	99946	157197	2161-07-14 20:29:00	2161-07-21 12:10:00	1	99946	1	2139-07-14 02:31:00	0	NaN
58962	99955	108494	2139-07-14 02:31:00	2139-07-19 05:20:00	1	99955	1	2157-02-20 22:56:00	0	NaN
58963	99957	148574	2157-02-20 22:56:00	2157-03-04 14:45:00	1	99957	1	2191-07-13 19:39:00	0	NaN
58964	99965	101083	2191-07-13 19:39:00	2191-07-16 12:08:00	1	99965	1	2191-08-23 23:16:00	0	NaN
58965	99966	167228	2191-08-23 23:16:00	2191-08-25 15:30:00	1	99966	1	2180-11-27 02:30:00	0	NaN
58966	99973	150202	2180-11-27 02:30:00	2180-12-01 13:42:00	1	99973	1	2156-11-28 11:56:00	0	NaN
58967	99982	151454	2156-11-28 11:56:00	2156-12-08 13:45:00	1	99982	3	2157-01-05 17:27:00	1	28.0
58968	99982	112748	2157-01-05 17:27:00	2157-01-12 13:00:00	2	99982	3	2157-02-16 17:31:00	0	35.0
58969	99982	183791	2157-02-16 17:31:00	2157-02-22 20:36:00	3	99982	3	2193-04-26 11:35:00	0	NaN
58970	99983	117390	2193-04-26 11:35:00	2193-04-29 13:30:00	1	99983	1	2181-01-27 02:47:00	0	NaN
58971	99985	176670	2181-01-27 02:47:00	2181-02-12 17:05:00	1	99985	1	2184-12-24 08:30:00	0	NaN
58972	99991	151118	2184-12-24 08:30:00	2185-01-05 12:15:00	1	99991	1	2144-07-25 18:03:00	0	NaN
58973	99992	197084	2144-07-25 18:03:00	2144-07-28 17:56:00	1	99992	1	2147-02-08 08:00:00	0	NaN
58974	99995	137810	2147-02-08 08:00:00	2147-02-11 13:15:00	1	99995	1	2117-12-30 07:15:00	0	NaN
58975	99999	113369	2117-12-30 07:15:00	2118-01-04 16:30:00	1	99999	1	2101-10-20 19:08:00	0	NaN